Skip to content

Adaptive loss-feedback loop + reactive ARQ - #3

Merged
vxfemboy merged 14 commits into
mainfrom
data-plane-feedback-arq
Jul 1, 2026
Merged

Adaptive loss-feedback loop + reactive ARQ#3
vxfemboy merged 14 commits into
mainfrom
data-plane-feedback-arq

Conversation

@vxfemboy

@vxfemboy vxfemboy commented Jul 1, 2026

Copy link
Copy Markdown
Member

Closes the data-plane control loop and activates the throughput win the throughput pass (PR #2) set up. Spec + plan committed; executed as 8 reviewed tasks (subagent-driven).

What landed

  • Feedback channel: an authenticated receiver→sender Control packet (PacketType::Control=3, sealed via the session AEAD) carrying {delivered_count, high_counter, missing[]}. New yip-transport modules: feedback (LossReport), lossdetect (gap-based LossDetector), retxbuf (RetxBuffer).
  • Class-aware controller: ARQ-eligible (Bulk) flows decay their repair ratio to zero on a clean link (firing the dormant FEC-encode bypass + halving datagrams); Realtime/Default keep a proactive floor. Loss snaps the ratio back up instantly.
  • Reactive ARQ: the sender retransmits NACKed Bulk objects with fresh RaptorQ repair symbols reusing the original object_id (tops up the receiver's existing decoder); Realtime/Default/expired objects are not retransmitted.

Results (release, netns, kernel 6.18 / Ryzen 5 7640U)

  • Throughput win proven: bulk repair ratio converges to 0.0000 (yipd diagnostic log); clean-link single-stream TCP ~273–285 → ~457 Mbit/s (~60%).
  • Loss recovery holds with the loop live: yip delivers 99.7% / 99.0% UDP at 5% / 10% netem loss (bare 94.7% / 89.9%).
  • Tunnel survives loss: netns ping 3/3 clean, 10/10 under 10% loss.

Review

Per-task reviews + a final whole-branch opus review. The final review found and we fixed one security must-fix (control-path on_seen now runs only after AEAD authentication, so a forged Control packet can't poison the loss detector) and raised yip-transport coverage to 90.05%. Cross-task counter semantics, ARQ object-identity (u16 wrap unreachable within the bounded window), whole-loop lock discipline (no deadlock, no lock across I/O), auth-before-trust, and bounded state all verified sound.

Follow-ups (documented, non-blocking): a dedicated end-to-end Bulk-ARQ integrity harness (establish tunnel before applying loss); the gap detector only reports fully-absent counters (exact for the single-symbol zero-repair case).

🤖 Generated with Claude Code

vxfemboy and others added 14 commits June 30, 2026 18:48
Closes the data-plane control loop: an authenticated receiver->sender Control
packet carrying delivered-count + missing object counters drives both (1) the
class-aware controller (bulk/default repair ratio earns zero on clean flows,
activating the dormant FEC bypass + halving datagrams; realtime keeps a floor)
and (2) deadline/class-aware reactive ARQ (sender retransmits fresh RaptorQ
repair symbols for eligible NACKed objects). Receiver reports raw gaps; sender
owns all class attribution and retransmit eligibility. Plan sequences feedback
first (throughput checkpoint), then ARQ.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8 tasks in two phases. Phase A (control channel + feedback + class-aware
zero-repair): LossReport (de)serialization, gap-based loss detector,
controller that lets ARQ classes earn ratio 0, daemon control-channel wiring,
and the Phase-A throughput verdict. Phase B (reactive ARQ): bounded retransmit
buffer, retransmit-on-NACK with fresh repair symbols reusing the original
object_id, plus the final measurement. Counter is one unified per-direction
sequence; sender owns class attribution; ARQ + zero-repair gated on FlowParams.arq.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement the feedback module for the receiver-to-sender loss report.
Includes serialization to big-endian wire format with 14-byte header
(delivered_count:u32, high_counter:u64, n_missing:u16) followed by
missing sequence numbers (u64 each). Decode validates input length and
rejects malformed bytes. Encode caps missing at MAX_NACK (64).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements LossDetector in crates/yip-transport/src/lossdetect.rs:
BTreeMap<u64,u64> pending set (counter → first-implied-ms), grace window
before declaring missing, window-bounded eviction of smallest keys.
Four tests green; full yip-transport suite 35/35; clippy clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both peers now send a sealed Control packet every 30 ms containing a
LossReport (delivered_count, high_counter, missing counters).  The
receiver decrypts the report, looks up each missing counter in a
bounded sent-log (counter → FlowClass, capacity 4096), computes
per-class loss fractions, and feeds them to Transport::observe_loss,
closing the feedback loop into the adaptive FEC controller.

Changes:
- handshake.rs: add PacketType::Control = 3
- tunnel.rs: SentLog ring (capacity 4096), LossDetector Arc<Mutex>,
  egress records counter→class in sent_log, ingress calls on_seen for
  every datagram (data and control) and on_delivered when FEC decodes,
  emits feedback every FEEDBACK_INTERVAL_MS=30, handles incoming
  Control packets to call observe_loss per class; periodic log of the
  bulk controller repair ratio every 5 s.
- yip-transport/lib.rs: add Transport::bulk_repair_ratio() for
  diagnostics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feedback loop converges the bulk controller to repair ratio 0.0000 (confirmed via
yipd diagnostic log), firing the FEC-encode bypass and halving per-packet
datagrams. Clean-link single-stream TCP rises from ~273-285 to ~457 Mbit/s (~60%),
consistent with removing the ~24us encode (80% of egress CPU) + 2->1 datagrams.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduces `RetxBuffer` in `crates/yip-transport/src/retxbuf.rs`: a
`HashMap<u64, Entry>` + `VecDeque<u64>` order list that holds at most
`max` ciphertext objects keyed by send-counter, evicting the oldest
entry (by insertion order) on cap overflow and skipping expired entries
(age > ttl_ms) on `get`. Mirrors the `FlowTable` pattern in `flow.rs`.
The stored `object_id: u16` is preserved so ARQ retransmits reuse the
receiver's existing FEC decoder for that object rather than starting a
new one.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- FecEncoder::repair_with_id: encode with an explicit object_id so the
  receiver's existing FEC decoder is topped up (not a new decode object).
- Transport::repair_object: thin wrapper used by the daemon.
- Unit test: retransmitted_repair_completes_a_missing_object (stall then
  top-up with extra_repair=4 symbols carrying the original object_id).
- tunnel.rs egress: after encode(), put (counter, ciphertext, class, oid)
  into the shared RetxBuffer (TTL 2s, cap 1024).
- tunnel.rs ingress control handler: after observe_loss, iterate
  report.missing; for each Bulk-class counter in the retx buffer, call
  repair_object and send the repair datagrams.  Lock discipline enforced:
  retx lock is never held across transport or io calls.
- run-netns-tunnel-loss.sh + ping_across_yipd_tunnel_under_loss: netem
  10% loss on both veth legs; 10 pings must all succeed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ve-repair rationale

The Default class emits a proactive repair symbol, so a 2-source object
encodes to 3 symbols; skipping one still decodes. Deliver exactly one
symbol to guarantee the object is genuinely incomplete before the
retransmit tops it up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Defense-in-depth hardening: LossReport::decode now caps the decoded
missing vec at MAX_NACK entries regardless of peer-supplied count,
bounding retransmit work per report. Added decode_caps_missing_at_max_nack
test that constructs raw bytes for a 100-entry report and verifies
successful decode with truncation to MAX_NACK (64).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase A throughput win holds (bulk ratio 0.0000, clean-link ~457 Mbit/s). Loss
recovery is unchanged with the feedback loop live: yip delivers 99.7%/99.0% UDP
at 5%/10% netem loss (controller re-arms FEC on loss). Reactive ARQ retransmits
Bulk NACKs with fresh repair symbols reusing the original object id; codec is
unit-proven, wiring reviewed, tunnel survives 10% loss. A dedicated end-to-end
Bulk-ARQ integrity harness is a tracked follow-up. Recorded in the bench README
and CHANGELOG.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… yip-transport coverage to >=90%

Fix 1 (security): in tunnel.rs ingress, the control-packet path called
detector.on_seen(counter, now_ms) BEFORE session.open(counter, ct),
letting a forged packet poison the loss detector's high_counter even
when authentication fails.  Move on_seen to run only inside the Ok
branch of session.open.  Data-path on_seen calls are unchanged.

Fix 2 (coverage): yip-transport line coverage was 89.53%; add four
focused unit tests in fec.rs covering repair_with_id, the late/duplicate
symbol guard (state.done), the SBN out-of-range guard for already-tracked
objects, and the full-Encoder encode path.  Line coverage: 90.05%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant